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Summary 


As the Army looks for more cost-effective, relevant, and timely means 
of delivering education and training, it has increasingly turned to dis¬ 
tributed learning. Historically, technology-based distributed learning 
consisted largely of single-learner, self-paced interactive multimedia 
instruction. However, such methods may not be effective for all types 
of training and education, particularly those that involve teamwork; 
group communication; critical thinking; and joint, interagency, inter¬ 
governmental, and multinational operations. The Command and Gen¬ 
eral Staff School’s (CGSS’s) Advanced Operations Course (AOC) is 
one example of a leader professional development course focused on 
Army Learning Model competencies. AOC focuses on developing 
junior field-grade staff officers’ skills, including the wide range of com¬ 
plex cognitive skills involved in planning military operations, with stu¬ 
dents working collaboratively as members of an operational and tacti¬ 
cal level staff. It is the second phase of Intermediate Level Education 
(ILE), building on the Common Core course. 

CGSS offers AOC using both a traditional resident model and a 
model that the school refers to as blended distributed learning (BDL). 
AOC-BDL requires substantial instructor-student and student-student 
interaction but is completely distributed. That is, the course uses a 
computer-supported cooperative learning (CSCL) model with exten¬ 
sive synchronous and asynchronous collaboration among students and 
instructors, who are distributed geographically and across time zones. 
When charged with providing AOC to all Army branch officers in all 
regions of the world without affecting the personnel operating tempo, 
CGSS adopted a BDL model as a flexible approach to achieve through- 
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put and meet course goals. However, relatively little is known about the 
effectiveness of using a completely distributed approach to support the 
acquisition of critical thinking skills and related competencies. 

The purpose of the study was to build on prior RAND Arroyo 
Center research on the effectiveness of the delivery venues for the ILE 
Common Core by examining outcomes for AOC-BDL. This study 
had three objectives: to assess AOC-BDL effectiveness, to identify best 
practices in CSCL, and to use findings from these efforts to identify 
options for improving AOC-BDL. To conduct the study, we analyzed 
responses to exit surveys measuring student satisfaction and perceptions 
of course quality and impact and piloted a survey of former graduates 
who completed the 2009-2010 AOC curriculum sometime in the past 
two years. We also conducted a review of research literature on blended 
learning and conducted original case studies of blended learning pro¬ 
grams in civilian higher education, military education, and industry. 

We adapted Benbunan-Lich, Hiltz, and Harasim’s (2005) online 
interaction learning model (see figure S.l) as a conceptual framework 
for this study. Benbunan-fich et al. posited that input factors, such 
as technology and course characteristics, influence learning processes, 
which in turn affect outcomes, such as learning and satisfaction. 

The main components of the framework are as follows: 

• Inputs: These comprise all the resources (including people) and 
contextual factors that contribute to the learning experience, 
including characteristics of technology, the course, instructors, 
students, and the organizational context. 

• Learning processes: These are the ways through which people 
learn, e.g., the types and extent of interactions between students 
and course content, instructors, and other students (Moore, 1989). 

• Outcomes: The framework includes the outcomes shown in 
figure S.l (learning effectiveness, scale or cost-effectiveness, 
access, student satisfaction, and faculty satisfaction). In this study, 
we focused primarily on two factors: learning effectiveness (as 
perceived by the students) and student satisfaction. 
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Figure S.1 

Framework for Online Learning 
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The Blended Distributed Learning Advanced Operations 
Course 

AOC-BDL was modeled after residential AOC, and the two venues 
are intended to be equivalent. Several course input factors are central 
to our analysis, conclusions, and suggested options for improvement in 
course design and delivery. These include the following: 

• Students. The course includes students from both the active 
component and the reserve component (RC). Most students 
work full time and have families. Students typically complete the 
course requirements in addition to their normal work or duty day 
requirements and on weekends; therefore, they have many com¬ 
peting demands on their own time. 

• Course content and structure. The course involves complex 
subject matter across multiple echelons and emphasizes peer col¬ 
laboration and development of staff products. AOC develops the 
officer’s abilities to analyze complex problems and recommend 
potential solutions and to build and ethically lead operational and 
tactical formations in a joint environment, among other goals. 
It has the equivalent of the 308 traditional classroom hours of 
instruction. To support a collaborative approach, AOC students 
are organized into 16-person groups called “staff groups,” each 
of which is composed of officers representing as broad a mix of 
branches and operational experience as possible. Although some 
of the course content is completed individually, using interactive 
multimedia instruction (IMI), readings, and written assignments, 
a significant portion of the course uses virtual synchronous activi¬ 
ties, including online classroom sessions (up to 20, two-hour ses¬ 
sions), and collaboration among students, to develop the staff 
products. Students are also required to participate in online 
threaded discussions about course topics. 

• Instructional technology. A range of technologies supports syn¬ 
chronous and asynchronous instruction, including Blackboard’s 
learning management system; Defense Connect Online (DCO), 
which is used for online synchronous interaction; IMI, also 
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referred to as computer-based instruction; simulations; and other 
communication media, such as email and telephone. Technolo¬ 
gies that are supported by CGSS run in a “dotmil” domain. 

• Policy. Department of the Army policy requires completion of 
the Common Core and AOC for promotion to lieutenant colonel 
in the active component and for promotion to colonel in the RC. 
Completion of both courses may also increase RC officers’ poten¬ 
tial for selection to many career-enhancing assignments. Unlike 
their colleagues in the resident course, students are not formally 
provided paid duty time to take AOC-BDL, and commanders are 
not required to allow students to take the course on duty time. 
Most students in this study reported working on discretionary 
time. 


Satisfaction and Perceived Learning Effectiveness in AOC- 
BDL 

In this section, we describe findings from a student survey adminis¬ 
tered at the end of AOC (“exit survey”) and the results of a pilot survey 
given to former AOC-BDL graduates (“postgraduate survey”). 

Students Were Generally Satisfied with the Course 

Responses to the exit survey indicate that nearly 80 percent of stu¬ 
dents felt that AOC-BDL achieved its core purpose. Most students 
gave favorable ratings to learning effectiveness for computer-based 
instruction and to operational topics, such as understanding joint force 
capabilities and limitations, the joint operational planning process, 
and change management processes. Most students also reported that 
they were satisfied with aspects of the learning environment, such as 
instruction quality and peer interaction and feedback, and they were 
generally satisfied with the course overall. 

However, 50 percent of students in the exit survey and 29 percent 
of students in the postgraduate survey reported that they would not 
recommend the course to others. Although we do not have conclusive 
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evidence about the reasons for these responses, some possible alterna¬ 
tives are addressed next. 

Students Noted Problems with Computer-Supported Collaboration 
for Learning 

Students provided many detailed responses to open-ended questions 
about course design and delivery. A common theme in these comments, 
not reflected in other survey questions, pertained to issues with com¬ 
puter-supported collaboration and learning. As shown in Figure S.2, 
the preponderance of comments focused on the CSCL approach. While 
some comments were positive, the majority were negative. A number of 
students felt that face-to-face interaction was needed, particularly for 
such topics as the military decision making process (MDMP); the most 
common recommendation was to have a one- to two-week in-person 
exercise at the end of the course. 

Students' Ratings of Course Effectiveness Suggest Improvement 
May Be Needed for Instruction of Some Course Topics 

While students were satisfied with instruction for some operational 
topics, as described above, their ratings may indicate needs for improve¬ 
ment in general critical field-grade competencies. Among former grad¬ 
uates, approximately 40 to 60 percent reported small or no improve¬ 
ment in conducting MDMP, complex problem solving, organizational 
leadership, and oral and written communication (see Figure S.3). In 
addition, although most reported that the course prepared them to lead 
in Army and joint environments, students felt less prepared for opera¬ 
tions in the international and interagency domains. 


Some Students Emphasized that the Completely 
Distributed Environment Posed Challenges to Meeting 
Course Goals 

Figure S.4 shows the negative sentiments about specific aspects of the 
CSCL approach. Most comments emphasized that the completely dis¬ 
tributed environment impeded some course learning goals, such as 
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Figure S.2 

Frequencies of Positive and Negative Comments—Exit Survey 
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Figure S.3 

Postgraduate Achievement of Critical Field-Grade Competencies 
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MDMP. Students also cited a variety of challenges to group coordina¬ 
tion in terms of technology reliability and access, competing demands 
on their time, complexity of the coursework, and the challenges of 
working across time zones. Likewise, a number of the respondents in 
the postgraduate survey commented that face-to-face interaction was 
needed for truly effective learning and that, while the available tech¬ 
nologies could support collaboration, conducting the MDMP process 
online was not beneficial. Although respondents reported being satis¬ 
fied with peer feedback, students in both surveys noted other prob¬ 
lems with collaboration, such as free riding (members not pulling their 
weight) and difficulties scheduling synchronous sessions with their 
teams. 

The Reliability of Technology Posed Problems for Some Students 

Approximately 30 to 40 percent of former graduates reported one or 
more problems with issues such as technical difficulties with computer- 
based instruction or reliable computer/Internet access. Not surpris- 


Figure S.4 

Specific Categories of Negative Comments About CSCL—Exit Survey 
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ingly, technical issues, such as Internet access, were more problematic 
among students who were located outside the continental United States 
during part or all of the course. Similar questions in the exit survey 
showed that 13 to 23 percent of respondents had trouble accessing a 
reliable computer; playing audio, video, or animations; or accessing the 
course over the Internet. CGSS staff confirmed that technical problems 
with DCO and Blackboard were common. 

Student Characteristics May Constrain Participation or Engagement 
in AOC-BDL 

Several characteristics of the student population present obstacles to 
working on the course. Students reported substantial work and family 
commitments and responded that these responsibilities interfered with 
working on the course (or that working on the course was a distrac¬ 
tion to performing their jobs). Only 14 percent of graduates reported 
that work or family commitments did not interfere with working on 
the course. 

Peer Collaboration, Interaction with Instructor, and Student 
Motivation Most Strongly Associated with Course Outcomes 

We conducted a series of analyses to examine associations among 
inputs, processes, and outcomes, where outcomes refers to overall sat¬ 
isfaction with the course and perceived learning effectiveness. There 
were several key findings. First, regarding input variables, whereas stu¬ 
dents’ ratings of technology reliability were not associated with out¬ 
comes, work and family commitments were negatively associated with 
satisfaction (but not with learning effectiveness). Second, student moti¬ 
vation for taking the course was strongly associated with satisfaction 
and perceived learning effectiveness. Graduates who took AOC-BDL 
to improve their performance or professionalism were more satisfied 
and reported greater learning than those who took the course for pro¬ 
motion or another reason (e.g., to avoid moving or because they were 
told the course was required). Third, students’ ratings of the quality of 
interaction with instructors and peers explained a substantial portion 
of outcome ratings beyond the effects of the input variables. Ratings 
of collaboration processes partially or fully mediated (accounted for) 
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the effect of inputs on outcome ratings. For example, work and family 
commitments had a negative effect on collaborative learning processes, 
which in turn affected satisfaction. These results highlight the impor¬ 
tance of collaborative processes in the course. 


Literature Review and Case Studies of BDL 

Both the literature review and case studies focused on blended learn¬ 
ing and distributed blended learning for adult learners in courses that 
address topics relevant to AOC (e.g., complex skills, collaboration). The 
literature review focused on empirical research published from 2007- 
2012, as well as seminal articles published prior to 2007; it also exam¬ 
ined studies of virtual team collaboration. Our original case studies 
included Pennsylvania State University’s (PSU’s) World Campus, which 
offers 80 degree or certificate programs; the Naval Postgraduate School 
(NPS), an accredited graduate-level education and research institution 
run by the U.S. Navy; and Xerox’s Service Delivery eXcellence, which 
teaches executives and managers service delivery and ways to solve 
service-related problems for clients and to expand business opportuni¬ 
ties. We supplemented these original case studies with findings from 
published cases. 

No Clear Advantage for Resident Instruction or Blended Learning, 
But There Are Few Examples of Effective All-Distributed BDL 

A number of studies have investigated the use of various instructional 
media for adult learners. Many of these studies compared resident 
instruction and distributed learning and, for the most part, found 
no clear advantage for one medium or the other. Some recent meta¬ 
analyses concluded that distributed learning has an advantage, but 
many of the studies included in these analyses have methodological 
weaknesses, such as selection biases. Moreover, there are few studies of 
effective all-distributed BDL. 

However, we did find a few examples of BDL relevant to AOC, 
including the award-winning Service Delivery eXcellence program 
at Xerox, which teaches leadership skills to executives and managers 
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around the globe; a pilot project to teach entrepreneurship education 
at three Canadian universities (Bisson et ah, 2005); and a field experi¬ 
ment comparing face-to-face and online courses for community psy¬ 
chologists in Italy (Francescato et ah, 2007). All three examples docu¬ 
ment successful delivery of BDL. 

Instructional Design, Not Venue, Is Key Determinant of Effectiveness 

Learning programs are effective when instructional strategies are 
designed to be compatible with human learning processes (Clark and 
Mayer, 2011). Thus, good instructional design will often differ accord¬ 
ing to the delivery medium used. Residential and distributed learn¬ 
ing designs can differ along many dimensions, including instructional 
approach, collaborative practices, feedback mechanisms, and grading 
strategies. 

Although scholars have concluded that blended learning and resi¬ 
dential learning require different instructional designs, many experts 
believe that effective blended learning dealing with complex mate¬ 
rial requires some degree of in-person instruction (e.g., Bernard et ah, 
2004). In fact, standard definitions of blended learning (which include 
some face-to-face instruction) and the scarcity of BDL programs 
and research may be indications of the perceived need for some in- 
person instructional activities, particularly when teaching such skills 
as leadership. 

New Technologies and Instructional Modalities Require Support to 
Be Effectively Adopted for Complex Learning 

Instructors are more likely to adopt a technology if they anticipate that 
it will be beneficial to student learning, if it is easy to use, and if they 
have institutional support for its use. Our case studies document how 
institutions can support technology following the recommendations 
of Park and Bonk (2007a), e.g., by providing professional development 
for online teaching and by having instructional designers and media 
specialists available to work with faculty on course development. In 
addition, our case studies revealed that commercial-off-the-shelf and 
open-source software for learning management systems and collabora¬ 
tive activities can successfully support BDL. 
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Successful Programs Find Ways to Reduce Conflicts with Students' 
Other Commitments 

Like AOC-BDL students, many adult learners lead busy lives and are 
geographically dispersed. We found that successful programs find ways 
to reduce conflicts with students’ other commitments. Programs use 
two general approaches to address time commitments and time-zone 
differences. First, they tend to limit the use of synchronous modalities, 
for example, by offering either distributed learning or blended learn¬ 
ing with a residential segment, but not BDL. This was the option the 
PSU programs preferred; directors and staff members said that they 
continue to use distributed learning rather than move to BDL because 
their students would resist the restrictions on their time imposed by the 
need to participate in synchronous online sessions. A second approach 
is to engage students’ employers to dedicate time for students’ course- 
work. For example, NPS requires that organizations employing pro¬ 
spective students agree to give students dedicated time for the course. 

Virtual and Collocated Groups Can Achieve Comparable Outcomes 
Under the Right Circumstances 

Numerous studies comparing outcomes of collocated and virtual 
groups show that collocated groups perform better on interdependent 
tasks, such as decisionmaking and problem solving, and express more 
positive outcomes, such as cohesion and trust, in the short term. How¬ 
ever, when time is not constrained or when virtual teams interact over 
time, virtual teams “catch up” to collocated teams in performance and 
social outcomes. Nonetheless, there are a variety of important processes 
in collocated teams that are difficult to replicate in distributed groups, 
and, as a result, virtual teams sometimes interact in ways that inhibit 
collaborative learning. 

Successful Programs Use a Variety of Methods to Foster Student 
Engagement and Success in Virtual Interactive Activities 

Case studies describe options instructors use to engage students and 
help them succeed in virtual collaboration activities. First, many 
instructors train students to ensure that they know how to use the 
technology. Second, instructors can “scaffold” discussions and provide 
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timely feedback. For example, in the NPS Program Management mas¬ 
ter’s program, instructors pose challenging questions in class or “cold 
call” on students to answer questions. Third, using small (rather than 
large) groups for interactive activities can alleviate free riding and facil¬ 
itate interaction and coordination. Fourth, peer evaluations are used to 
increase accountability in teams. 


Conclusions and Options for Improvement 

AOC-BDL Has a Number of Strengths 

In summary, AOC-BDL has a number of strengths. Despite signifi¬ 
cant constraints, most students report that AOC-BDL meets its core 
purpose. Furthermore, students give high ratings to items about the 
importance of student-instructor and student-student interaction, and 
students are consistently satisfied with their instructors. Students also 
report that most computer-based instruction lessons are effective. These 
indicators of success are more impressive in light of the fact that many 
students do not have a choice about the medium; that is, they cannot 
choose whether to attend the resident course or the BDL course. In 
contrast, educational institutions often give students a choice about 
enrolling in a resident or DL course; as a result, the relative effective¬ 
ness of these methods is often difficult to determine. CGSS’s continu¬ 
ous improvement process is yet another strength in that it gives the 
school many opportunities to make needed adjustments to the course. 

In addition to these strengths, participation in AOC-BDL pro¬ 
vides some benefits that students in the resident course do not expe¬ 
rience. Notably, by collaborating in virtual teams on complex tasks, 
students in AOC-BDL learn to work in situations that are increas¬ 
ingly common in operational environments and in other institutional 
settings. 

Improvements Can Be Made 

Despite these strengths, student responses indicate factors to consider 
in CGSS’s continuous improvement process for AOC-BDL. Students’ 
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ratings of learning effectiveness suggest the need for further investiga¬ 
tion of instruction of critical field-grade competencies and leadership 
in a range of operational environments. For example, it would be ben¬ 
eficial to obtain input from students’ commanders before and after the 
course with respect to knowledge, skills, and abilities in these areas. 

More important, changes are needed to foster better collabora¬ 
tion among students to facilitate learning and coordination. Currently, 
AOC-BDL has a number of input factors in conflict, including com¬ 
plex subject matter, requirements for collaboration at a distance, use of 
technologies that are time consuming for collaboration, and students 
who have limited time available to devote to the course. Options for 
improvement in course design and delivery focus on alleviating the 
tensions among these factors. 

Improving Course Design and Delivery 

We present a number of options for improvement in course design and 
delivery, summarized in Table S.l. These options are based on an inte¬ 
gration of survey results with best practices identified in case studies 
and the research literature. Suggestions address input factors including 
course design, student characteristics, technology, and organizational 
policy. 

Modify Course Design 

Adding a resident segment to the class offers the greatest potential for 
improvement in learning processes and outcomes and is consistent with 
student recommendations and best practices in military and civilian 
higher education. However, this change is also the most complicated 
in terms of scheduling and resourcing. Indeed, this option may not be 
feasible because of available infrastructure, costs, and operational man¬ 
ning requirements associated with temporary duty and course staffing. 
In light of these constraints, we argue that the remaining strategies 
become particularly important to consider. 

A second option is to shift some collaborative activities to higher- 
level IMI (e.g., Level 3) and have fewer but more in-depth synchro¬ 
nous exercises, with more time spent reviewing plans, discussing the 
rationale for students’ decisions, and evaluating alternative courses of 
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Table S.1 

Summary of Options to Improve Course Design and Delivery 


Strategy 

Suggestion 

Course design 

Add a 1- to 2-week resident period at the end of the course to 
work on the most intensive collaborative activities 


Shift some collaborative activities to higher-level IMI or other 
assignments with iterative student-instructor interaction; have 
fewer but more in-depth synchronous exercises 


Implement peer evaluations 


Offer additional instructor facilitation to student groups and 
train students in team facilitation skills 

Composition of 
student groups 

Use smaller groups for team assignments and for larger group 
exercises 


Compose groups based on time zone 

Organizational 

policy 

Encourage chain of command or employers to provide dedicated 
time for training 

Technology 

Move to a dotcom domain 


Pilot alternative technologies for group collaboration and group 
online gaming and simulations 


action. This approach might free some instructional resources, but IMI 
can be costly to develop and maintain. Alternatively, students could be 
given complex analytic individual assignments with iterative student- 
instructor interaction to foster complex thinking skills. 

To increase accountability, CGSS could implement peer evalu¬ 
ations that would contribute to course grades. Better facilitation for 
student groups as they work on their assignments (from instructors, or 
by training the students themselves to facilitate) could also reduce free 
riding. These options are low cost and could be executed in the near 
term. 

Change Composition of Student Groups 

Using smaller groups, such as eight-person rather than 16-person staff 
groups for some exercises, would reduce free riding and alleviate some 
of the coordination challenges that students reported. CGSS should 
emphasize to instructors the importance of composing groups based 
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on time zone, and if a particular area of expertise needed for a group 
assignment is not available in a close time zone, providing a “stand-in” 
to serve in this role. 

Affect Organizational Policy 

It may be possible to reduce conflicts with job commitments by engag¬ 
ing students’ employers (including commanders). This would entail 
finding ways to encourage commanders to provide duty time for 
required training. For RC students who work full time, CGSS could 
inform employers when an employee is taking the course, state expec¬ 
tations about what the student is required to do, and ask for support 
to allow the student to spend time on the course. Given that many of 
the skills addressed in AOC, such as problem solving, communication, 
and working in distributed teams, are relevant to a wide variety of jobs, 
the school might gain buy-in from employers by informing them how 
they can benefit from their employee’s participation in the course. 

Change Technology to Fit Course Design 

Moving the course to a dotcom or dotedu domain could potentially 
alleviate some technology reliability issues that may inhibit access to 
the course. Hosting the course and tools for collaboration on a dotcom 
can also provide more options for collaborative technologies. For these 
reasons, Blackboard.com is being piloted with some staff groups, and 
CGSS anticipates moving to a dotcom domain entirely by 2014 (see 
Gould, 2013). A second suggestion is to continue to explore new tech¬ 
nologies, including software for collaboration, as reported in our case 
studies, and technologies relevant to course content, such as online 
group games or simulations. 

Expand Course Evaluation to Better Support a Continuous 
Improvement Process 

Future evaluations of ILE courses should use a broader range of 
approaches. Although student assessments of their learning are often 
correlated with actual learning, survey responses are subjective. Future 
evaluations of AOC and other ILE courses should also analyze objec¬ 
tive measures of learning, such as grades on assignments and tests. 
Because many of the competencies addressed in AOC are abstract, it 
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may be helpful to provide information to students to demonstrate their 
knowledge gain use a pretest-posttest approach. 

It is important to note that the exit and postgraduate surveys were 
administered only to students who completed the course. In the future, 
CGSS should also assess the perceptions of those who drop out and 
their reasons for attrition. This omission is a key gap, given that the 
attrition rate is currently estimated at 25 percent. As a result of that 
high rate, survey responses likely overestimate the satisfaction and per¬ 
ceived learning effectiveness of the course. More important, the attri¬ 
tion rate significantly increases the cost of the course per graduate and 
keeps waiting lists unnecessarily long. Assessing reasons for attrition in 
more depth could lend additional urgency to the changes already sug¬ 
gested in this report or could lead to additional improvements in the 
course that could reduce the dropout rate. 

Using multiple methods and data sources is important to provide 
information about different aspects of the course. Other sources of data 
for evaluation include 

• data from commanders regarding graduates’ job performance fol¬ 
lowing AOC completion 

• interviews or focus groups with students or faculty to obtain 
detailed feedback about the learning experience and recommen¬ 
dations for improvement 

• Web-based analytics to investigate student use of online materials 

• independent subject-matter experts’ observation of class sessions. 


Concluding Thoughts 

This study identified important options to consider when training 
complex material using group collaboration that is entirely distrib¬ 
uted. Because AOC-BDL objectives and instructional methods sup¬ 
port Army Learning Model goals and because the course has many 
positive features, this blended learning approach should be considered 
for other courses. However, alternative approaches to collaboration are 
needed when course activities involve collaboration on complex mate- 
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rial. CGSS’s experience and continuous development and evaluation 
efforts put the school in a position to identify such improvements and 
inform the Army training community about use of BDL for education 
and training. 


